Systems and methods for e-commerce API orchestration using natural language interfaces

ABSTRACT

Systems and methods for e-commerce systems using natural language understanding are described. A computing device is configured receive a user utterance and identify at least one semantic element within the user utterance. An intent associated with the at least one semantic element is identified and an intent flow associated with the identified intent is executed. The intent flow includes a set of tasks executed in a predetermined order. A system utterance is generated by instantiating a response template selected from a plurality of response templates associated with the executed intent.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 16/177,296, entitled “SYSTEMS AND METHODS FORE-COMMERCE API ORCHESTRATION USING NATURAL LANGUAGE INTERFACES,” filedon Oct. 31, 2018, now U.S. Pat. No. 11,238,850, the disclosure of whichis hereby incorporated by reference in its entirety.

TECHNICAL FIELD

This application relates generally to natural language interfaces, andmore particularly, to natural language-based e-commerce interfaces.

BACKGROUND

Natural language processing refers to the ability of a system, such as acomputer system, to recognize and interpret human speech. Naturallanguage processing can understand user utterances (e.g., spoken usersentences, phrases, etc.) and can include steps such as speechrecognition, syntax processing, semantic processing, and discoursegeneration. The use of natural language processing allows users tointeract with systems without the need to use specialized input devices(e.g., keyboard, mouse, etc.). Natural language processing has becomethe de facto input method for many internet-enabled and/orinternet-of-things (IoT) devices, such as digital assistants (e.g.,Siri, Cortana, Google Home, Alexa, etc.) and smart devices.

Although, current natural language systems are designed to provideresponses to general queries and handle certain specific tasks, currentsystems provide very limited contextual recall and make conversationssound robotic. Current systems provide unnatural responses, are not ableto reference prior utterances to identify known information, and are notable to identify previously provided contextual information. Further,current digital assistants and smart devices exist in “walled gardens”in which each digital assistant or smart device provides certaincapabilities only for systems using the identical software and/orhardware.

SUMMARY OF THE INVENTION

In various embodiments, a system including a computing device isdisclosed. The computing device is configured to receive a userutterance and identify at least one semantic element within the firstuser utterance. The computing device is further configured to identifyan intent associated with the at least one semantic element and executean intent flow associated with the identified intent. The intent flowincludes a set of tasks executed in a predetermined order. A systemutterance is generated by instantiating a response template selectedfrom a plurality of response templates associated with the executedintent.

In various embodiments, a method is disclosed. The method includesreceiving a user utterance and identify at least one semantic elementwithin the user utterance. An intent associated with the at least onesemantic element is identified and an intent flow associated with theidentified intent is executed. The intent flow includes a set of tasksexecuted in a predetermined order. A system utterance is generated byinstantiating a response template selected from a plurality of responsetemplates associated with the executed intent.

A non-transitory computer readable medium having instructions storedthereon, wherein the instructions, when executed by a processor cause adevice to perform operations including receiving a user utterance andidentify at least one semantic element within the user utterance. Theprocessor identifies an intent associated with the at least one semanticelement is and an intent flow associated with the identified intent isexecuted. The intent flow includes a set of tasks executed in apredetermined order. A system utterance is generated by instantiating aresponse template selected from a plurality of response templatesassociated with the executed intent.

BRIEF DESCRIPTION OF THE DRAWINGS

The features and advantages will be more fully disclosed in, or renderedobvious by the following detailed description of the preferredembodiments, which are to be considered together with the accompanyingdrawings wherein like numbers refer to like parts and further wherein:

FIG. 1 illustrates a block diagram of a computer system, in accordancewith some embodiments.

FIG. 2 illustrates a networked environment configured to implement avoice-based e-commerce environment, in accordance with some embodiments.

FIG. 3 illustrates a voice-based e-commerce environment including anatural language conversation architecture configured to providevoice-enabled e-commerce services, in accordance with some embodiments.

FIG. 4 illustrates a natural language conversation architectureimplemented by the voice-based e-commerce environment of FIG. 1 , inaccordance with some embodiments.

FIG. 5 illustrates a sentence parsed by a natural language understandingcomponent of the natural language conversation architecture of FIG. 4 ,in accordance with some embodiments.

FIG. 6A illustrates a context model data structure configured tomaintain a context stack including user and system utterances, inaccordance with some embodiments.

FIG. 6B illustrates a context entry of the context stack of FIG. 6A, inaccordance with some embodiments.

FIG. 7 is a flowchart illustrating a method of intent-entity mining anddialogue stack growth, in accordance with some embodiments.

FIG. 8 is a flowchart illustrating a method of context mining, inaccordance with some embodiments.

FIG. 9 is a state transition diagram illustrating an intent handler ofthe natural language conversation system of FIG. 4 , in accordance withsome embodiments.

FIG. 10 is an example state transition diagram illustrating a task flowgraph of the intent handler of the natural language conversation systemof FIG. 4 , in accordance with some embodiments.

FIG. 11 illustrates a multi-tenant user flow for authoring responsesusing the intent handler of the natural language conversation system ofFIG. 4 , in accordance with some embodiments.

FIG. 12 illustrates a response generation component of the naturallanguage conversation system of FIG. 4 , in accordance with someembodiments.

FIG. 13 illustrates a response generation branch visualization forgenerating a response for a search intent, in accordance with someembodiments.

DETAILED DESCRIPTION

The ensuing description provides preferred exemplary embodiment(s) onlyand is not intended to limit the scope, applicability or configurationof the disclosure. Rather, the ensuing description of the preferredexemplary embodiment(s) will provide those skilled in the art with anenabling description for implementing a preferred exemplary embodiment.It is understood that various changes can be made in the function andarrangement of elements without departing from the spirit and scope asset forth in the appended claims.

FIG. 1 illustrates a computer system 2 configured to implement one ormore processes, in accordance with some embodiments. The computer system2 is a representative device and may include a processor subsystem 4, aninput/output subsystem 6, a memory subsystem 8, a communicationsinterface 10, and/or a system bus 12. In some embodiments, one or morethan components of the computer system 2 can be combined or omitted suchas, for example, omitting the input/output subsystem 6. In someembodiments, the computer system 2 includes one or more additionalcomponents not shown in FIG. 1 , such as, for example, a powersubsystem. The computer system 2 may include two or more instances ofone or more of the components 4-12 shown in FIG. 1 . For example, thecomputer system 2 may include two or more memory subsystems 8. For thesake of conciseness and clarity, and not limitation, one of each of thecomponents is shown in FIG. 1 .

In various embodiments, the processor subsystem 4 includes processingcircuitry operative to perform one or more functions of the computersystem 2. The processor subsystem 4 may be implemented as a generalpurpose processor, a chip multiprocessor (CMP), a dedicated processor,an embedded processor, a digital signal processor (DSP), a networkprocessor, an input/output (I/O) processor, a media access control (MAC)processor, a radio baseband processor, a co-processor, a microprocessorsuch as a complex instruction set computer (CISC) microprocessor, areduced instruction set computing (RISC) microprocessor, a very longinstruction word (VLIW) microprocessor, and/or other suitable processingdevice. The processor subsystem 4 may also be implemented by acontroller, a microcontroller, an application specific integratedcircuit (ASIC), a field programmable gate array (FPGA), a programmablelogic device (PLD), and so forth.

In various embodiments, the processor subsystem 4 is arranged to run anoperating system (OS) and/or various applications. Examples of an OSinclude, for example, operating systems generally known under the tradename of Apple OS, Microsoft Windows OS, Android OS, Linux OS, and anyother proprietary or open source OS. Examples of applications include,for example, network applications, local applications, data input/outputapplications, user interaction applications, natural language processingapplications, e-commerce applications, etc.

In some embodiments, the computer system 2 includes a system bus 12 thatcouples various components including, for example, the processingsubsystem 4, the input/output subsystem 6, and/or the memory subsystem8. The system bus 12 can be any of several types of bus structure(s)including a memory bus or memory controller, a peripheral bus orexternal bus, and/or a local bus using any variety of available busarchitectures including, but not limited to, 9-bit bus, IndustrialStandard Architecture (ISA), Micro-Channel Architecture (MSA), ExtendedISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB),Peripheral Component Interconnect Card International Association Bus(PCMCIA), Small Computers Interface (SCSI) or other proprietary bus,and/or any custom bus suitable for computing device applications.

In some embodiments, the input/output subsystem 6 includes one or moremechanisms or components to enable a user to provide input to thecomputer system 2 and for the computer system 2 to provide output to theuser. For example, the input/output subsystem 6 may include a button,keypad, keyboard, click wheel, touch screen, motion sensor, microphone,camera, etc. In some embodiments, the input/output subsystem 6 includesa visual peripheral output device for providing a display visible to theuser. The visual peripheral output device may include a screen such as,for example, a Liquid Crystal Display (LCD) screen. As another example,the visual peripheral output device may include a movable display orprojecting system for providing a display of content on a surface remotefrom the computer system 2. The input/output subsystem 6 may furtherinclude an audible output device, such as a speaker, configured toconvert received signals to audible output. In some embodiments, theinput/output subsystem 6 includes coder/decoders, also known as Codecs,to convert digital media data into analog signals. For example, thevisual peripheral output device may include video Codecs, audio Codecs,or any other suitable type of Codec.

In some embodiments, the communications interface 10 includes hardwareand/or software capable of coupling the computer system 2 to one or morenetworks and/or additional devices, such as the network 50 illustratedin FIG. 2 . The communications interface 10 may be arranged to operatewith any suitable technique for controlling information signals using adesired set of communications protocols, services or operatingprocedures. The communications interface 10 may include the appropriatephysical connectors to connect with a corresponding communicationsmedium, whether wired or wireless, such as a wired and/or wirelessnetwork.

In various aspects, the network may include local area networks (LAN) aswell as wide area networks (WAN) including without limitation Internet,wired channels, wireless channels, communication devices includingtelephones, computers, wire, radio, optical or other electromagneticchannels, and combinations thereof, including other devices and/orcomponents capable of/associated with communicating data. For example,the communication environments can include various devices, and variousmodes of communications such as wireless communications, wiredcommunications, and/or combinations of the same.

Wireless communication modes include any mode of communication betweenpoints (e.g., nodes) that utilize, at least in part, wireless technologyincluding various protocols and combinations of protocols associatedwith wireless transmission, data, and devices. Wired communication modesinclude any mode of communication between points that utilize wiredtechnology including various protocols and combinations of protocolsassociated with wired transmission, data, and devices. In variousimplementations, the wired communication modules may communicate inaccordance with a number of wired protocols. Examples of wired protocolsmay comprise Universal Serial Bus (USB) communication, RS-232, RS-422,RS-423, RS-485 serial protocols, FireWire, Ethernet, Fibre Channel,MIDI, ATA, Serial ATA, PCI Express, T-1 (and variants), IndustryStandard Architecture (ISA) parallel communication, Small ComputerSystem Interface (SCSI) communication, or Peripheral ComponentInterconnect (PCI) communication, to name only a few examples.

Accordingly, in various aspects, the communications interface 10 mayinclude one or more interfaces such as, for example, a wirelesscommunications interface, a wired communications interface, a networkinterface, a transmit interface, a receive interface, a media interface,a system interface, a component interface, a switching interface, a chipinterface, a controller, and so forth. When implemented by a wirelessdevice or within wireless system, for example, the communicationsinterface 10 may include a wireless interface having one or moreantennas, transmitters, receivers, transceivers, amplifiers, filters,control logic, and so forth.

In various aspects, the communications interface 10 may provide datacommunications functionality in accordance with a number of protocols.Examples of protocols may include various wireless local area network(WLAN) protocols, including the Institute of Electrical and ElectronicsEngineers (IEEE) 802.xx series of protocols. Other examples of wirelessprotocols may include various wireless wide area network (WWAN)protocols, such as GSM cellular radiotelephone system protocols withGPRS, CDMA cellular radiotelephone communication systems with 1×RTT,EDGE systems, EV-DO systems, EV-DV systems, HSDPA systems, etc. Furtherexamples of wireless protocols may include wireless personal areanetwork (PAN) protocols, such as an Infrared protocol, a protocol fromthe Bluetooth Special Interest Group (SIG) series of protocols, etc. Yetanother example of wireless protocols includes near-field communicationtechniques and protocols, such as electro-magnetic induction (EMI)techniques, such as passive or active radio-frequency identification(RFID) protocols and devices. Other suitable protocols may include UltraWide Band (UWB), Digital Office (DO), Digital Home, Trusted PlatformModule (TPM), ZigBee, and so forth.

In some embodiments, at least one non-transitory computer-readablestorage medium is provided having computer-executable instructionsembodied thereon, wherein, when executed by at least one processor, thecomputer-executable instructions cause the at least one processor toperform embodiments of the methods described herein. Thiscomputer-readable storage medium can be embodied in memory subsystem 8.

In some embodiments, the memory subsystem 8 includes machine-readable orcomputer-readable media capable of storing data, including bothvolatile/non-volatile memory and removable/non-removable memory. Thememory subsystem 8 may include at least one non-volatile memory unitcapable of storing one or more software programs. The software programsmay contain, for example, applications, user data, device data, and/orconfiguration data, or combinations therefore, to name only a few. Thesoftware programs may contain instructions executable by the variouscomponents of the computer system 2.

In various aspects, the memory subsystem 8 may include read-only memory(ROM), random-access memory (RAM), dynamic RAM (DRAM), Double-Data-RateDRAM (DDR-RAM), synchronous DRAM (SDRAM), static RAM (SRAM),programmable ROM (PROM), erasable programmable ROM (EPROM), electricallyerasable programmable ROM (EEPROM), flash memory (e.g., NOR or NANDflash memory), content addressable memory (CAM), polymer memory (e.g.,ferroelectric polymer memory), phase-change memory (e.g., ovonicmemory), ferroelectric memory, silicon-oxide-nitride-oxide-silicon(SONOS) memory, disk memory (e.g., floppy disk, hard drive, opticaldisk, magnetic disk), or card (e.g., magnetic card, optical card),and/or any other type of machine-readable media.

In some embodiments, the memory subsystem 8 includes an instruction set,in the form of a file for executing various system components, such ascomponents of a voice-enabled e-commerce platform, as described herein.The instruction set may be stored in any acceptable form of machinereadable instructions, including source code and/or various appropriateprogramming languages. Some examples of programming languages that maybe used to store the instruction set include, but are not limited to:Java, C, C++, C#, Python, Objective-C, Visual Basic, .NET programming,etc. In some embodiments a compiler or interpreter is configured toconvert the instruction set into machine executable code for executionby the processing subsystem 4.

FIG. 2 illustrates a networked environment 50 configured to enablenatural language e-commerce, in accordance with some embodiments. Thenetworked environment 50 includes at least one user system 52 a, 52 b,at least one e-commerce system 54, at least one designer system 56, atleast one engineer system 58, and/or any other suitable systems. Each ofthe systems 52-58 can include a computer system, such as the computersystem 2 described above in conjunction with FIG. 1 . It will beappreciated that each of the systems 52-58 can include generic systemsand/or special purpose systems, and are within the scope of thisdisclosure.

In some embodiments, each of the systems 52-58 are configured toexchange data over one or more networks, such as network 60. Forexample, in some embodiments, the user system 52 (and/or any othersystem) is configured to receive speech input from a user and providethe received speech input to the e-commerce system 54. The e-commercesystem 54 is configured to provide speech-enable e-commerce services, asdiscussed in greater detail below with respect to FIGS. 3-13 . In someembodiments, the e-commerce system 54 implements a natural languageconversation architecture to receive speech input data, extract anintent and/or one or more entities, execute one or more identifiedintents (or subtasks), and/or generate an appropriate response. In someembodiments, the natural language conversation architecture can includea natural language understanding component, a context mining component,an intent handler, and/or a response generator, as discussed in greaterdetail below. Although embodiments are discussed herein includingspecific systems and/or configurations, it will be appreciated that thenetworked environment 50 can include any number of systems, can combineone or more of the identified systems, and/or can include additional oralternative systems, in various embodiments.

FIG. 3 illustrates a speech-enabled e-commerce environment 100 includinga natural language conversation architecture 102 configured to providespeech-enabled e-commerce services, in accordance with some embodiments.The speech-enabled e-commerce environment 100 includes at least one usersystem 52 configured to receive speech input from a user and convert thespeech input into one or more computer-readable signals. Each usersystem 52 includes a speech recognition front-end configured to receiveaudio signals from a user, for example via a microphone, and convert thereceived audio into computer-readable digital signals. The speechrecognition front-end can include a speech-enabled assistant, aspeech-enabled application, generic speech-recognition softwareintegrated with one or more applications, smart devices, and/or anyother suitable speech recognition front-end.

For example, in various embodiments, a user system 52 can include atleast one speech-enabled assistant 106, such as Amazon Alexa, Siri,Google Home, Microsoft Cortana, etc., and/or any other suitable voiceand/or speech enabled assistant. The speech enabled assistant 106 isconfigured to receive speech input from a user, convert the speech inputto digital signals, and may be configured to perform one or more actionsin response to recognized speech in the speech input. In someembodiments, the speech-enabled assistant 106 is configured tocommunicate with the natural language e-commerce architecture 102 toprovide a speech-enabled e-commerce experience to a user, as describedin greater detail below.

As another example, in some embodiments, a user system 52 can include aspeech-enabled application 108, such as an e-commerce applicationassociated with a specific retailer, platform, product, vendor, etc. Thespeech-enabled application 108 is configured to receive speech inputfrom the user and provide speech data to the natural languageconversation architecture 102.

As yet another example, in some embodiments, a user system 52 caninclude general purpose speech recognition software 110 integrated withone or more additional applications or devices. For example, a usersystem 104 can include voice recognition software such as one or more ofthe Dragon line of products available from Nuance Communications, Inc.,one or more of the Fluency line of products available from MModal, Inc.,Voice Finger, PenPower ViaTalk, Tatzi, e-speaking, and/or any othersuitable speech-recognition software. The speech recognition software110 is integrated with one or more additional programs implemented bythe user system 104, such as, for example, web-based applications,mobile application, smart device applications, and/or any other suitablesoftware. Integration between the general purpose voice recognitionsoftware 110 and at least one additional program, such as a web browser,provides a user front-end for interaction with the natural languageconversation architecture 102.

As still another example, in some embodiments, a user system 52 caninclude a speech-enabled smart device 112 having one or more circuitsand/or software programs configured for speech recognition. As usedherein, the term “smart device” is used to denote any electronic and/orpartially electronic device configured to operate interactively and/orat least partially autonomously. In some embodiments, a speech-enabledsmart device 112 may include one or more speech-enabled assistants, oneor more special purpose speech recognition circuits, and/or any othersuitable speech recognition system. For example, in various embodiments,the user system 104 can include a smart speaker, such as, but notlimited to, Amazon Echo, Google Home, Sonos One, Apple HomePod, etc. Asanother example, in various embodiments, the user system 104 can includeany suitable speech-enabled device, such as a smart television, smartappliance, smart home hub, etc.

In some embodiments, an e-commerce system 54 provides a natural languageconversation architecture 102 including a natural language applicationprogramming interface (API) 114 configured to receive and/or providespeech data and/or other data to and from the user system 52, e.g., toand from the speech-recognition front-end. As discussed in greaterdetail below, in some embodiments, the natural language conversationarchitecture 102 includes a plurality of adapters 122 a-122 c (see FIG.4 ) each configured to communicate with one or more predeterminedspeech-recognition front-ends. Each adapter 122 a-122 c converts thereceived speech and/or other data into a common format and provides thereceived data to the conversation API 114, which communications with oneor more additional elements of the natural language conversationarchitecture 102 to provide a speech-enabled e-commerce platform.Similarly, each adapter 122 a-122 c is configured to receive speechoutput from the natural language conversation architecture 102 andconvert the speech output into a form suitable for the user system 52.

In some embodiments, the natural language conversation architecture 102includes a natural language understanding (NLU) engine 130, an intenthandler 135 (including a context mining component 140 and a taskstitching component 150), and a response generation component 160. TheNLU engine 130, the intent handler 140, and the response generatorcomponent 160 cooperate to provide a conversational e-commerceinteraction with a user through the user system 104. For example, insome embodiments, the natural language conversation architecture 102 isconfigured to receive natural language input from a user (via the usersystem 52, speech-enabled front end, and conversation API 114), extractan intent and/or one or more entities from the user speech, execute anidentified intent, and/or generate one or more responses in response tothe processed intent.

Each user utterance received by the natural language conversationarchitecture 102 may include an intent and/or one or more entitiesassociated with an intent. As used herein, the term intent refers to anaction or task that the user wants the system to complete. For example,in an e-commerce environment, possible intents may include, but are notlimited to, an add-to-cart intent to add one or more products to a usercart, a product information request intent, a cart information requestintent, a clarification intent, a cancel intent, a shipping informationrequest intent, a complete purchase intent, a modify cart intent, etc.As one example, a conversation between a user and the natural languageconversation architecture 102 of an e-commerce environment 100 mayinclude the following utterances:

-   -   User: Add apples to cart. [1]    -   System: How many? [2]    -   User: How many can I add? [3]    -   System: Ten. [4]    -   User: Ok, add four. [5]    -   System: Ok, added four apples to your cart. [6]        Utterance [1], provided by the user, includes an “add-to-cart”        intent, e.g., the user wishes to add apples to their cart. The        NLU engine 130 parses the user utterance and identifies the        add-to-cart intent. The NLU engine 130 further identifies an        entity associated with the “add to cart” action, i.e., a product        (apples). As used herein, the term entity refers to a variable        or facet of a user-requested intent that is required to complete        an action. For example, examples of entities in an e-commerce        environment include, but are not limited to, a count entity, a        size entity, a brand entity, a product entity, a price entity, a        domain entity, a department entity etc.

In some embodiments, each required entity for executing an intent isincluded in a single utterance. When a required entity is not present inan utterance including an intent, the NLU engine 130 identifies themissing entities and invokes the intent handler 135 and the responsegenerator component 160 a to identify the missing entities in priorutterances and/or generate a slot filling response to request themissing entity. In the above example, the natural language conversationarchitecture 102 generates a slot filling request to identify how manyapples should be added to the cart, statement [2] above. The slotfilling response is provided to the natural language API 114, whichprovides the response to the speech-enabled frontend of the user system52. The user system 52 converts the response into machine-spoken audiofor the user.

After generating a slot filling response, the natural languageconversation architecture 102 expects the user to respond with anutterance including a quantity entity. However, in the above example,the user responds with an utterance [3] containing a new intent, i.e., aproduct information intent. The NLU engine 130 identifies the productinformation intent in utterance [3], but does not identify a productentity (which is required to execute the product information intent),i.e., the user has omitted the product apples. The NLU engine 130determines that a product entity is missing from utterance [3] andinvokes a context mining element 140 (for example, through a contextmining plugin) to review previous utterances in the conversation andattempt to identify the missing entity, as discussed in greater detailbelow. In the above example, the context mining element 140 identifies aproduct identification in statement [1] of the conversation,specifically, apples. After identifying the missing entity, the naturallanguage conversation architecture 102 a invokes the intent handle 135to determine a execute a task to identify how many apples a user can addto a cart and invokes the response generation component 160 to generatea response including an answer to the user's query, i.e., utterance [4].

The user responds with utterance [5], e.g., “add four.” The NLU engine130 identifies the intent of the statement, i.e., an “add-to-cart”intent. As discussed in greater detail below, in some embodiments, thenatural language conversation architecture 102 is configured toassociate the add-to-cart intent in utterance [5] with the prioradd-to-cart intent in utterance [1]. The NLU engine 130 furtheridentifies a required entity of the add-to-cart intent, specifically, aquantity of four. However, the NLU engine 130 determines that a secondrequired entity, i.e., a product entity, is missing from utterance [5].As discussed above, the natural language conversation architecture 102 autilizes the context mining component 140 to review prior utterances andidentify missing entities that were provided in the prior utterances. Inthe instant example, the context mining component 140 identifies aproduct, apples, in utterance [1]. After identifying all requiredentities, the intent handler 135 is invoked to process the add-to-cartintent according to a predetermined intent flow generated by a designerand/or engineer, as discussed in greater detail below. After completingthe add-to-cart intent, the response generation component 160 generatesa success response indicating the add-to-cart intent was executedsuccessfully, i.e., statement [6].

In some embodiments, the natural language conversation architecture 102includes a plurality of knowledge domains 116 a-116 c configured toprovide domain-specific APIs 117, a domain-specific knowledge base 118(e.g., intents, entities, interface elements, inventory, speech models),domain specific flows 119 (e.g., dialogue flows, task flows, intentflows) and/or other domain-specific data. For example, in variousembodiments, the plurality of knowledge domains 116 a-116 c can includephysical product knowledge domains such as domains associated withgroceries, home goods, sporting goods, etc., digital product knowledgedomains such as domains associated with digital videos, music, books,etc., company/vendor specific knowledge domains associated with aspecific company, line of products, vendor, etc., department specificknowledge domains associated with specific collections of productsdenoted by a retailer such as pharmacy products, in-store products,on-line only products, etc., and/or any other suitable knowledgedomains. In some embodiments, the natural language conversationarchitecture 102 utilizes domain-specific data within each of theplurality of knowledge domains 116 a-116 c to process received speechinput, identify intents and/or entities included in the speech input,execute one or more intents or tasks, and/or generate one or moreresponses, each of which is discussed in greater detail below.

FIG. 4 illustrates a system diagram of one embodiment of the naturallanguage conversation architecture 102 a, in accordance with someembodiments. The natural language conversation architecture 102 a issimilar to the natural language conversation architecture 102 discussedin conjunction with FIG. 3 , and similar description is not repeatedherein. In some embodiments, the natural language conversationarchitecture 102 a includes a natural language API 114 a including oneor more adapters 122 a-122 c. Each adapter 122 a-122 c is configured tointeract with a predetermined speech-enabled frontend, such as, forexample, a speech-enabled assistant. In the illustrated embodiment, afirst adapter 122 a is configured to interact with a firstspeech-enabled assistant such as, for example, Google Home, a secondadapter 122 b is configured to interact with a second speech-enabledassistant, such as, for example, Microsoft Cortana, and a third adapter122 c is configured to interact with a third speech-enabled assistant,such as, for example, Apple Siri. Although each of the illustratedadapters 122 a-122 c is configured to interact with a specificspeech-enabled assistant, it will be appreciated that the conversationAPI 114 a can include any number of adapters 122 a-122 c configured tointeract with any number of speech-enabled frontends, such as, forexample, speech-enabled assistants, programs, systems, devices, etc.

In some embodiments, each adapter 122 a-122 c is configured to convertplatform-specific data (such as speech data) to data formatted for useby the natural language conversation architecture 102 a. For example, insome embodiments, each adapter 122 a-122 c is associated with a specificspeech-enabled assistant having a predetermined set of capabilities,functions, response, etc. The use of adapters 122 a-122 c makes thenatural language conversation architecture 102 a platform-agnostic andallows the use of multiple third party platforms, programs, systems,devices, etc., without needing to update the entire natural languageconversation architecture 102 a for each new third party speech-enabledsystem.

After conversion, the now platform-agnostic speech input is provided toa talk controller 162 a and further to a plugin executor 164 configuredto execute one or more plugins 160 a-160 d. In some embodiments, theplugins 160 a-160 d include one or more business rule plugins 160 a-1160c, an NLU adapter plugin 160 d, a reference resolver plugins 160 e, anintent processor plugins 160 f, and/or any other suitable plugins. Theplugin executor 164 is configured to execute each of the plugins 160a-160 d in a predetermined order. For example, the plugin executor 164may be configured to execute each of the plugins 160 a-160 f in apredetermined sequential (i.e., serial) order, a predetermined paralleland/or partially parallel order, and/or any other suitable order. Aplugin flow (i.e., order of execution) may be stored in and/or retrievedfrom a plugin flow database 165. In some embodiments, the plugin flow istailored to a predetermined environment, such as an e-commerceenvironment.

In some embodiments, the business rule plugins 160 a-160 c areconfigured to apply one or more rules to an input prior to providing theinput to one or more additional plugins 160 d-160 f. The business ruleplugins 160 a-160 c can be configured to provide rules, work flows,and/or other processing specifications for a predetermined domain and/orenvironment. For example, in the illustrated embodiment, the firstbusiness rule plugin 160 a is configured to provide natural languageprocessing rules for an e-commerce environment 100, such asretailer-specific, environment-specific, and/or other predeterminedrules. The business rules may include one or more intents, associatedentities, and/or other elements for interaction with the naturallanguage conversation architecture 102 a. Similarly, the second instanceof the business rule plugin 160 b may provide domain-specific rules forexecuting one or more intents and/or tasks identified by the NLU engine130 a and/or for generating one or more domain-specific ordomain-modified responses. The third instance of the business ruleplugin 160 c may provide domain-specific and/or environment-specificrules for post-intent and/or post-response tasks.

In some embodiments, the NLU adapter plugin 160 d is configured toinvoke an NLU Engine 130 a. The NLU engine 130 a is configured toperform natural language processing and understanding on received speechinput (e.g., as recognized by the user system 104 and/or the naturallanguage API 114) to generate a computer-readable representation of thecontent of the user's speech. In some embodiments, the NLU engine 130 ais configured to identify and parse user utterances to identifypredetermined components of the user utterance. For example, asillustrated in FIG. 5 , the NLU engine 130 a may be configured toidentify semantic entities (or components) 132 a-132 e within a singleuser utterance 134, such as, for example, a “count” component 132 a, a“size” component 132 b, a “brand” component 132 c, a “product type”component 132 d, an “intent” component 132 e, and/or any other suitablecomponents. Although specific embodiments are discussed herein, it willbe appreciated that the components can include any suitable componentsidentified by a business rule, speech model, etc., such as, for example,based on specific market segments, departments, and/or any othersuitable grouping of products or services provided by an e-commerceplatform.

As illustrated in FIG. 5 , the NLU engine 130 a is configured toidentify both single-word and multiple-word semantic components. Forexample, in the illustrated embodiment, the NLU engine 130 a identifieseach of a count component, a size component, a brand component, and aproduct type component that are a single word or number. In addition,the NLU engine 130 a identifies an intent component 132 e that includesthree words. In some embodiments, one or more semantic components, suchas the intent component 132 e, can include multiple words, numbers, etc.that are continuous and/or discontinuous within the user utterance 134.Each of the semantic components 132 a-132 e can include single-elementand/or multi-element semantic components.

In some embodiments, after parsing and identifying the semanticcomponents 132 a-132 e within the user utterance 134, the naturallanguage conversation architecture 102 a is configured to identify auser intent. For example, the natural language conversation architecture102 a can be configured to identify a user request to perform one ormore actions, such as searching for one or more products within ane-commerce environment, adding one or more products to a cart,completing a purchase, receiving shipping information, etc. In someembodiments, the natural language conversation architecture 102 a usesone or more data structures configured to store and/or identify contextof one or more user utterances.

In some embodiments, the NLU engine 130 a includes a machine-learningnatural language engine configured to provide natural languageunderstanding based on one or more machine-learning generated speechmodels. The speech models are configured to combine a deterministic(finite state automation (FSA)) models and neural-net models to generatea final speech recognition model configured to provide semantic parsingof user utterances. The speech models may be generated by an NLUtraining and annotation tool 137 configured to receive a set of trainingdata and one or more machine-learning models. The NLU engine 130 agenerates one or more speech models 136 a-136 c according to a trainingprocess and the training data stored in a parsing database 139.

In some embodiments, the NLU engine 130 a generates and/or receives aplurality of domain-specific speech models 136 a-136 c configured toprovide domain-specific vocabularies, semantic parsing, and/orstructures to the NLU engine 130 a. In some embodiments, one of thedomain-specific speech models 136 a-136 c is selected based on one ormore rules provided by a business rule plugin 160 a prior to invokingthe NLU engine 130 a. In some embodiments, the NLU engine 130 a may beaccessed through one or more input methods, such as, for example, a webservice 141.

In some embodiments, after parsing the user utterance, a referenceresolver plugin 132 e is configured to invoke a context mining component140 a to resolve any explicit and/or implied references in a userutterance. Context mining is the process of traversing a conversationalcontext of a given user or user-system interaction in order to improveunderstanding of a most-recent user utterance. For example, in someembodiments, the context mining component 140 a is configured toretrieve specific information about and/or infer a relationship betweenspecific entities or intents in a user-system conversation. In someembodiments, a data structure including a sequentially-storedstructured-representation of user and system utterances in aconversation provides a contextual record of what was said, by who, andwhen in a conversation. In some embodiments, the data structure includesmetadata relevant to the conversation (or dialogue exchange) between theuser and the natural language conversation architecture 102 a. Forexample, an add-to-cart intent may require a predetermined set ofentities for execution. If each of the entities has been provided in aprior utterance, the context mining component 140 a identifies themissing entities and invokes the task stitcher 150 a to execute theintent without requiring additional input from the user.

FIG. 6A illustrates a context model data structure 200 configured tomaintain a context stack 202 including user and system utterances, inaccordance with some embodiments. The context stack 202 includes aplurality of context entries 204 a-204 c, each corresponding to anintent identified by the NLU engine 130 a in a user utterance. Forexample, each context entry 202 a-202 c can include an intent associatedwith a specific environment such as intents associated with ane-commerce environment, including, but not limited to, an add to cartintent, a search intent, a product information intent, and/or any othersuitable intent. Each context entry 204 a-204 c within a context stack202 includes a predetermined structure, as illustrated in FIG. 6B.

As shown in FIG. 6B, in some embodiments, each context entry 204includes a root intent 206. The root intent 206 includes an intentspecified in a user utterance. In some embodiments, each context entry204 includes an entity list 208 that identifies the required entitiesassociated with the root intent 206. For example, for an add-to-cartintent, the entity list 208 may include a product entity and a quantityentity. In some embodiments, the entity list 208 may further includeoptional entities. For example, to continue the add-to-cart intentexample, the entity list 208 may further include a brand entity, a sizeentity, a price entity, etc. Although specific embodiments are discussedherein, it will be appreciated that the entity list can include anysuitable entities related to the intent. Intents and associated entitiesmay be specified by, for example, one or more business rules generatedby a designer and/or engineer for a selected domain. The entity list 208is updated as each required and/or optional entity is identified in anutterance.

In some embodiments, each context entry 204 includes a dialogue stack210 including a plurality of dialogue frames 212 a-212 d representativeof a user utterance or a system utterance in a conversation. In theillustrated embodiment, the context entry 204 includes two dialogueframes 212 a, 212 c representative of user utterances and two dialogueframes 212 b, 212 d representative of system-generated utterances. Inthe illustrated embodiment, the plurality of dialogue frames 212 a-212 dare maintained in chronological order within the dialogue stack 210,although it will be appreciated that the dialogue frames 212 a-212 d canbe stored in any predetermined and/or searchable order.

In some embodiments, each of the dialogue frames 212 a-212 d include oneor more dialogue elements 214-220 such as an intent element 214identifying an intent of a user utterance 212 a, 212 c, an entity setelement 216, identifying a set of entities located a user utterance 212a, 212 c (for example, quantity, type, brand, product, etc.), an answeras list element 218 configured to maintain a plurality of responsiveinformation (e.g., products, prices, etc.) in a list format for latterrecall, and a response type element 220 configured to identify the typeof the system utterance, such as, for example, a slot filling response,a success response, an error response, a product information responseetc. In some embodiments, the intent element 214 and/or an element inthe entity set 216 may be provided by a prior utterance.

FIG. 7 is a flowchart illustrating a method 300 for intent-entity miningand dialogue stack growth, in accordance with some embodiments. At step302, a user utterance 301 is received. The NLU engine 130 a processesthe received user utterance 301 and to identify an intent within theuser utterance 301. If the NLU engine 130 a identifies an intent, themethod 300 proceeds to step 304. If the NLU engine 130 a fails toidentify an intent, the method 300 proceeds to step 306.

At step 304, the NLU engine 130 a attempts to identify required entitiesfor executing/completing the identified intent. For example, in variousembodiments, an intent in the e-commerce space may require one or moreof a product entity, a size entity, a quantity entity, a brand entity,or other entities associated with e-commerce transactions. If each ofthe required identities is provided in the user statement 301, themethod 300 proceeds to step 308. If at least one required identity isnot identified in the user statement 301, the method 300 proceeds tostep 314.

At step 308, a new context entry 204 is generated and added to a contextstack 202 associated with the current user. The root intent 206 of thenew context entry 204 is set as the intent identified at step 302, eachentity identified at step 304 in added to the entity list 208, and adialogue frame 210 a-210 d representative of the user utterance 301 isadded to the dialogue stack 210. After generating the new context entry204, the method 300 proceeds to step 310 and executes the intent. Tocontinue the example from above, if the identified intent includes is anadd-to-cart intent, the natural language conversation architecture 102 aexecutes the add-to-cart intent to add the identified product to thecart. The method 300 proceeds to step 312 and generates a successresponse, as discussed in greater detail below with respect to FIGS.12-13 . The success response indicates a successful execution of theidentified intent to the user.

As discussed above, if the user utterance 301 does not include at leastone required entity, the method 300 proceeds to step 314. At step 314,the context mining element 140 a review dialogue frames 212 a-212 d inthe dialogue stack 210 of the most-recently generated context entry 204to identify the one or more missing entities. In some embodiments, thecontext mining element 140 a searches through each dialogue frame 212a-212 d in the dialogue stack 210 in reverse-chronological order (i.e.,most recent dialogue frame first) to attempt to identify the missingentity or entities. At step 316, the context mining element 140 adetermines whether the missing entity or entities have been identified.If all of the missing entities are identified in the dialogue stack 210,the method 300 proceeds to step 318, and the natural languageconversation architecture 102 a adds a dialog frame 212 a-212 drepresentative of the user utterance 301 to the dialogue stack 210 ofthe currently active context entry 204 and proceeds to step 310, whichis discussed above.

If at least one entity remains unidentified, the method 300 proceeds tostep 320. The context mining component 140 a generates a new contextentry 202 a-202 c. The generated context entry 202 a-202 c includes thecurrently identified intent as a root intent, any entities required toexecute the intent in the entity list, and places the user statement 210e into the dialogue stack of the generated context entry 202 a-202 c.The method proceeds to step 322 and generates a slot filling response.The slot filling response prompts the user to provide the missingentities to complete the intent. For example, if the user intent is “addto cart” and the missing entities include a quantity, the naturallanguage conversation architecture 102 a, using the task stitcher 150 aand the response generator 160 a, may generate a slot filling responseof “how many” to prompt the user to provide a quantity entity. Althoughspecific examples are discussed herein, it will be appreciated thatresponse generator 160 a can generate any suitable slot fillingresponse, as discussed in greater detail below.

As discussed above, if an intent is not identified at step 302, themethod 300 proceeds to step 306, and the natural language conversationarchitecture 102 a determines if any entities are identified in the userutterance 301. As discussed previously, potential entities included in auser utterance 301 a may be include environment-specific entities (e.g.,an e-commerce environment may have a first set of expected entities),domain-specific entities (e.g., a grocery domain may have a second setof expected entities), and/or any other suitable entity context. If theNLU engine 130 a identifies one or more entities, the method 300proceeds to step 324 and reviews the dialog stack 210 and/or the contextstack 202 for an intent that matches the entity or entities identifiedin the user utterance 301. At step 326, the natural languageconversation architecture 102 a determines if a matching intent wasidentified. If a matching intent is identified, the method 300 proceedsto step 318, discussed above. If a matching intent is not found at step326 or no entities are identified at step 306, the method 300 proceedsto step 328 and generates a failure response to indicate the failure ofthe natural language conversation architecture 102 a to understand theuser utterance 301.

In some embodiments, a context model data structure 200 maintains aformal representation of conversational context, including a pluralityof variables or functions configured to represent a status of theconversation and/or a user utterance. For example, in some embodiments,a formal representation of conversation context may include, but is notlimited to, a set of variables and/or functions including:

-   -   I={i₁, i₂, . . . , i_(n)}, representative of the set of all        intents;    -   E={e₁, e₂, . . . , e_(n)}, representative of the set of all        entities;    -   ReqEnt(i), representative of the set of entities needed to        process an intent i, where i ϵ I    -   AllEnt(i), representative of a set of all entities that are        relevant for intent i, where i ϵ I. ReqEnt(i) ⊆ AllEnt(i)    -   Needs(e)={i: e ϵ ReqEnt(i)}, representative of a set of all        intents for which e is required    -   RelevantFor(e)={i: e ϵ AllEnt(i)}, representative of a set of        all intents for which e is relevant    -   UU=(i, Es, Ts), representative of a User Utterance, e.g., a        triplet of the intent i (ϵ I), some entities Es (⊆E), and        timestamp Ts    -   CU=(rt, rd, Ts), representative of a System Utterance, e.g., a        triplet of a reply type rt, reply data rd and timestamp Ts    -   rt ϵ {terminal:success, terminal:failed, nonterm:question:        (boolean|set):(boolean_target|set): (closed|open)}    -   rd=[(e1, v1), . . . ], a, possibly empty, list of entity names        and entity values

In some embodiments, the context model data structure 200 may augmentuser and/or system utterances to add additional context or information.For example, and without limitation, a unique utterance id (uid) may begenerated for each utterance, an utterance may include a reference to aprior utterance (reference to prior uid), an intent may be modified witha Boolean identifying whether the intent has been completed, etc. Insome embodiments, the references to prior uids (e.g., crefs) aregenerated by the context mining component 140 a during a context miningprocess. In some embodiments, the natural language conversationarchitecture maintains a conversation context initiated by a userutterance and completed by a system utterance, e.g.:

-   -   Context=UU (UU|CU)*CU

In some embodiments, each user utterance may be classified using one ormore interaction variables. In some embodiments, the interactionvariables are configured to model the type of user utterance received.For example, in some embodiments, the set of interaction variablesapplied to each user utterance may include, but is not limited to:

-   -   A: UU.i≠ϕ, which identifies whether an intent is present in UU.        Boolean. A=1 if yes, else A=0    -   B: ReqEnt(UU.i)⊆UU.Es, which identifies whether all entities        required by UU.i are present in UU.Es. Boolean. B=1 if yes, else        B=0    -   C: UU.Es≠ϕ, which identifies whether any entities are present at        all in UU. Boolean. C=1 if yes, else C=0    -   D: cref jump<=1, which identifies whether UU refers to just the        previous CU or to nothing at all. Boolean. D=1 if yes, else D=0    -   E: cref→CU, which identifies whether cref refers to a CU.        Boolean. E=1 if yes, else E=0=>refers to UU or nothing at all    -   F: cref type: Set        -   when E=1 (=>referring to CU)            -   F=0: Context[cref].rt=question            -   F=1: D=1 and Context[cref].rt≠question            -   F=2: D=0 and USING Context[cref].rd.E            -   F=3: D=0 and REFERRING to Context[cref].rd.E        -   when E=0 (=>referring to UU or nothing at all)            -   F=0: EDIT UU[cref].Es            -   F=1: USE UU[cref].Es            -   F=2: INFER UU[−1].i using UU[cref].i (infer open intent)            -   F=3: NULL no reference.                The disclosed variables may be used to classify and/or                process each received user utterance according to a                predetermined intent-entity mining process. For example,                using the above example, each user utterance may be                represented as a sequence of 5 binary digits (e.g.,                values of A-E) and one quaternary digit (e.g., value of                F). In some embodiments, multi-turn utterances provide                context for interpreting a user utterance and                representing the user utterance according to the                disclosed models. For example, in some embodiments, two                user utterances may be considered fundamentally                different if their representations according to the                above model are different. An example of a conversation                including classifications according to the disclosed                model is provided below:    -   UU>Add apples to cart (101103)    -   CU>Sure, I can get you gala apples, sold individually for 58        cents.    -   How many do you need?    -   UU>What's the maximum that I can add (100113)    -   CU>24    -   UU>Ok add 10 (001010)    -   CU>Ok added 10 gala apples to your cart    -   UU>By the way, are they rich in iron (100113)    -   CU>Yes apples are very rich in iron. 1 serving of apple, which        is usually x grams, satisfy y % of your daily iron needs    -   UU>Ok then let's do 15 (001000)    -   CU>Ok changed your cart to make it 15 apples

FIG. 8 is a flowchart illustrating a classification 400 of utterancesused by the context mining component 140 a according to the modeldiscussed above, in accordance with some embodiments. A first node 402represents a set of all utterances that may be received by the naturallanguage conversation architecture 102 a. Node 402 is subdivided intonode 404 representing a set of incomplete utterances and node 406representing a set of complete utterances 406. Node 404 may be furtherdivided into a leaf 408 representing a set of utterances including anintent but missing at least one required entity and a node 410representing a set of utterances missing an intent. Node 410 issubdivided into a node 412 representing a set of utterances having atleast one required entity present and a leaf 414 representing set ofutterances lacking both an intent and an entity. Node 412 is subdividedinto leaf 416 representing a set of utterances that are an answer to apreviously posed question, a leaf 418 representing a set of utterancesthat are a re-specification of entities already mentioned, and leaf 420representing a set of utterances that require intent inference.Similarly, node 406 is subdivided into a leaf 422 representing a set ofutterances having a result reference and a leaf 424 representing a setof utterances not having a result reference.

In some embodiments, a context mining algorithm may be applied at eachnode and/or leaf within the classification 400 illustrated in FIG. 8 .For example, in some embodiments, a specific context mining processand/or portion of a context mining process may be applied at each nodeof the categorization scheme 400 to sort a received user utterance intoan appropriate leaf and apples a predetermined process at each leaf Insome embodiments, the natural language conversation architecture 102 aapplies a predetermined logic check against variables A and B todetermine a sub-node for continued processing and sorting of a userutterance. For example, in the illustrated embodiment, the naturallanguage conversation architecture 102 a applies a logic check of: ifA==1 and B==1, go to node 406, else go to node 404. That is, the naturallanguage conversation architecture 102 a checks to see if both an intentand all required entities are present.

If both A and B are set to 1, indicating the presence of both intent andrequired entities, the context mining process proceeds to node 406 anddetermines whether the current utterance refers to a previous utterance.For example, in some embodiments, given a complete user utterance (i.e.,A==1, B==1), the natural language conversation architecture 102 a checksthe context stack to determine if the identified intent of the currentstatement is located within the context stack and is in an open state,indicating a reference to a previous utterance. If the current userutterance refers to a prior utterance, the context mining processclassifies the user utterance in leaf 422 and the user utterance isprocessed by a previous result handling process that generates tasksand/or responses with knowledge of the referred result by the taskstitcher 150 a and the response generator 160 a, as discussed in greaterdetail. If the current user utterance does not reference a priorutterance, the user utterance is classified in leaf 424 and the userutterance is processed according to a normal intent processing flow, asdiscussed in greater detail below.

Referring back to node 402, if the context mining process determinesthat at least one of A and B are set to 0, the context mining processproceeds to node 404. At node 404, the context mining process checks thevalue of A to determine whether an intent is present. If A==1(indicating an intent is present but at least one required entity ismissing), the user utterance is classified in leaf 408. In someembodiments, when a user utterance is classified in leaf 408, thecontext mining element 140 a is configured to review the dialogue stack210, up to predetermined number of frames 212 a-212 d, to determine ifthe one or more missing entities have been previously identified. If themissing entities are identified within the dialogue stack 210, the taskstitcher 150 a processes the current intent using the identifiedentities. If at least one necessary entity is not identified in thedialogue stack 210, the task stitcher 150 a and/or the responsegenerator 160 a generate a slot filling response to prompt the user tosupply the missing entities.

If, at node 404, the context mining process determines A==0 (indicatinga missing intent), the context mining process proceeds to node 410 anddetermines whether the user utterance includes at least one entity. Forexample, in the illustrated embodiment, the context mining processchecks the value of variable C to identify the presence (C==1) of atleast one entity or the absence (C==0) of any entity. If C==0 (e.g., noentities identified), the context mining process classifies the userutterance in leaf 414. In some embodiments, the task stitcher 150 aand/or the response generator 160 a are configured to generate an errorresponse when a user utterance is classified in leaf 414, indicating theuser utterance cannot be processed.

If, at node 410, the context mining process determines if at least oneentity is identified (C==1), the context mining process proceeds to node412 and determines the type of user utterance that was received. Forexample, in some embodiments, the context mining element 140 a firstdetermines if the dialogue stack 210 contains an open slot fillingquestion for the entity included in the user utterance. If the contextmining element 140 a identifies an appropriate open slot fillingquestion, the user utterance is classified in leaf 416 and handled bythe task stitcher 150 a as a slot filling intent.

If the context mining element 140 a does not identify an appropriateopen slot filling question, the context mining process determines if theuser utterance is a re-specification of one or more previously providedentities. For example, in some embodiments, the context mining element140 a determines if the context stack contains an intent that wasrecently executed (e.g., the intent was executed within a predeterminednumber of dialogue frames 212 a-212 d or within a predetermined timeperiod). If a recent intent is identified, the context mining element140 a determines whether the recent intent can be re-executed with theentity received in the current utterance. If the recent intent can bere-executed, the content mining element 140 a classifies the userutterance in leaf 418 and invokes the task stitcher 150 a to re-executethe intent with the updated entity information.

If a recent intent is not identified, or an identified intent cannot bere-executed with the provided entity, the context mining processdetermines if an intent can be inferred from the entity or entities inthe user utterance For example, in some embodiments, the context miningelement 140 a examines the context stack 202 to determine if a recent oropen intent in the context stack 202 matches the identified entity inthe user utterance. If the entity matches an open intent, the contextmining element 140 a classifies the user utterance in leaf 420 andprocesses the user utterance in conjunction with the identified openintent. If the context mining element 140 a is unable to associate theuser utterance with an open intent, an error message is generated by thetask sticher 150 a and/or the response generator 160 a. Although aspecific embodiment of a context mining process and associatedclassifications has been disclosed herein, it will be appreciated thatthe context mining process can include additional, fewer, and/oralternative leafs and/or nodes, the leafs/nodes can be rearranged,and/or alternative or additional processing can be performed at eachnode, and is within the scope of this disclosure.

With reference again to FIG. 4 , in some embodiments, after the contextmining element 140 a classifies the user utterance, a intent processorplugin 132 f invokes a task stitcher 150 a to execute an intent or taskflow associated with an intent identified by the NLU engine 130 a and/orthe context mining element 140 a. In various embodiments, the taskstitcher 150 a is configured to execute a plurality of intent flowsusing granular and reusable processing blocks to implement one or morepredefined task flows. The intent flows and/or the task flows may beprovided by one or more designers and/or engineers, as discussed ingreater detail below. In some embodiments, the intent flow and/orassociated task flows can be generated from a subsets of the reusableprocessing blocks based on one or more rules associated with a specificenvironment, domain, intent, etc. For example, in the illustratedembodiments, the second instance of the business rule plugin 160 b maybe configured to provide intent and/or task specific flows to the taskstitcher 150 a for executing an identified intent.

The task stitcher 150 a provides a configurable user experience byallowing designers and/or engineers to add and/or modify processingblocks. The task stitcher 150 a decouples the intent processing logicfrom the the NLU engine 130 a and provides multi-tenant support. In someembodiments, the task stitcher 150 a includes a plurality of intentflows each including a subset of the processing blocks and configured toexecute a specific intent within a specific domain.

FIG. 9 is a finite state machine 500 illustrating an intent flowimplemented by a task stitcher 150 a, in accordance with someembodiments. The steps for executing a specific intent are divided intoa number of states 502-508 and state transitions 510 a-510 c by the taskstitcher 150 a, for example, based on an intent flow provided to thetask stitcher 150 a. Each state 502-508 is representative of a task thatis executed by the task stitcher 150 a to execute an identified intent.Each task (e.g., state 502) accepts a set of input data and produces aset of output data and one of a predetermined set of states 504-508. Theselected output state 504-508 is provided as an input to the statemachine 500 and determines which task (e.g., state) is then executed.The task stitcher 150 a is configured to identify the task flowspecifications for the identified intent and effect the necessary statechanges to process each task in the intent flow. In some embodiments, hetask stitcher 150 a operates as a task executor service and providesinput/output between tasks and/or provides data at the edges (e.g.,initial input or final output).

FIG. 10 illustrates an add-to-cart task flow 550 implemented by the taskstitcher 150 a, in accordance with some embodiments. The add-to-carttask flow 550 receives a customer identification (CID) entity and,optionally, a product entity. At state 552, the task stitcher 150 aexecutes a task to determine if a product entity (e.g., a requiredentity identifying the product) was provided in the input data. If theproduct entity was provided, the task stitcher 150 a proceeds to state554, which invokes response generation task to enable the responsegenerator 160 a to generate a slot filling request for the missingproduct entity. After executing the response generation task 554, taskstitcher 150 a completes processing of the current intent and thenatural language conversation architecture 102 a waits for a userresponse to the slot filling response. A new or existing intent isidentified based on the user response and is executed by the taskstitcher 150 a.

If the product identity is present, the task stitcher 150 a proceeds tostate 556 and executes a task to determine if a quantity entity (e.g., arequired entity identifying the quantity of the product entity) ispresent. The quantity entity may only be required for certainadd-to-cart task flows. If the quantity entity is not present, the taskstitcher 150 a proceeds to task 554, which executes a responsegeneration task to generate a slot filling request for the quantityentity. It will be noted that the same state 554 is used to execute theresponse generation task for generating the slot filling request for amissing product identity and a missing quantity entity, as state 554 isa reusable processing block. The output of the response generation taskexecuted at state 554 is dependent on the input provided by the taskstitcher 150 a as generated by the state 552, 556 immediately precedingthe transition to state 554.

If the quantity entity is present, the task stitcher 150 a transitionsto state 558 and executes a task to determine if a price entity (e.g., arequired entity identifying the cost of each unit of the product entity)is present. If the cost entity is present, the task stitcher 150 atransitions to state 560 and executes an add-to-cart task to add theproduct (identified by the product entity) at the quantity (quantityentity) and price (price entity) to the user's cart (CID entity). Thetask stitcher 150 a proceeds to state 562 and executes a responsegeneration task to generate a success response.

If the price entity is not present, the task stitcher 150 a proceeds tostate 564 and executes a search task to search for the price of theproduct, for example, within a system database associated with productsin a specific domain, store, website, etc. If the price entity isidentified during the search task 564, the task stitcher 150 a isproceeds to task 560 and executes the add-to-cart task, as discussedabove. If a price entity is not identified, the task stitcher proceedsto state 562 and executes a response generation task to generate anerror response indicating that the requested add-to-cart intent couldnot be completed. As with state 554, it will be noted that the responsegeneration task executed at state 562 is dependent on the output of thepreceding state 560, 564.

FIG. 11 illustrates a multi-tenant user flow 600 configured to providemulti-tenant response authoring for a task stitcher 150 b, in accordancewith some embodiments. In some embodiments, multiple user interfacedesign systems 56 a, 56 b interact with a task flow authoring system 604to generate one or more intent flows and/or task flows to be implementedby the task stitcher 150 a. The intent/task flows can include aplurality of processing blocks stitched together to provide apredetermined user experience based on an identified intent, task,domain, environment, input state, etc. In some embodiments, theinterface designer systems 56 a, 56 b are limited to a predetermined setof processing blocks generated by one or more engineer systems 58 a, 58b. The engineer systems 58 a, 58 b interact with the task generationsystem 610 to generate processing blocks for specific domains,environments, etc. The generated processing blocks may be stored in adatabase 612 containing task flows, intent flows, etc. and theassociated processing blocks. Each of the designer systems 56 a, 56 baccess a flow repository 614 a, 614 b associated with a specific domainand can generate intent flows from the predetermined task flows in thedatabase 612. The intent flows and associated task flows are implementedby a task flow loader 620 when an intent processing request is receivedby an intent processing plugin 132 f. A task execution service 624 isconfigured to provide feedback to the database 612 regarding successfuland unsuccessful task/intent executions such that the designer systems56 a, 56 b and/or the engineer systems 58 a, 58 b can update thetask/intent flows.

FIG. 12 illustrates a response generation component 160 a of the naturallanguage conversation system of FIG. 4 , in accordance with someembodiments. In some embodiments, the response generation component 160a is configured to execute a generate response task. The responsegeneration component 160 a is configured to apply one or more rules,knowledge bases, etc. to select and instantiate a response template fora selected response type. For example, in an e-commerce environment, theresponse generation component 160 a may be configured to generate a slotfilling response (e.g., a request for a missing entity), a successresponse (e.g., indicate the identified intent has been performed,provide requested information), an error response (e.g., indicate theidentified intent cannot be executed), and/or any other suitableresponse. In various embodiments, the response generation component 160a provides a voice-friendly, frictionless interface that generates apolymorphic, varied, configurable, and multi-modal output to a userbased on the user system 52 a, 52 b, one or more stored responsetemplates, domain-specific rules, environment specific rules, etc.

In some embodiments, the response generator component 160 a isconfigured to execute a response generation task 702. The responsegeneration task 702 receives an input including a response type,entities (or facets) required or optional for executing the intent, andcapabilities of the user system 52 a, 52 b. The response generation task702 normalizes 704 each entity to a standard format identified by aresponse database 710. The response database 710 can include a pluralityof entity normalization rules 712 designed by an interface designer 602a for a specific domain. For example, in various embodiments, theplurality entity normalization rules 712 may include rules forgenerating one or more versions of a product name, rules for generatingprices (e.g., stating twenty-five cents as opposed to 0.25 dollars),rules for generating the sound of the spoken response (e.g., tone)

After normalizing the received entities, the response generation task702 randomly selects 706 a response template from a plurality ofresponse templates associated with the identified response type. Forexample, in some embodiments, the database 710 includes a plurality ofadd-to-cart response templates 714, a plurality of search responsetemplates 716, a plurality of product info response templates 718,and/or any other suitable response templates. Each set of templates714-718 includes one or more speech templates that are appropriate forresponding to the associated intent (e.g., add-to-cart, search, query,etc.). The set of normalization rules 712 and/or the sets of responsetemplates 714-718 can be generated and/or updated by a user interfacedesigner system 56 and stored in the response database 710 by a dataloader and refresher task 720 that updates the response database 710. Asan example, a set of add-to-cart response templates may include, but isnot limited to:

-   -   Ok, I've added [SHORT TITLE] to your cart, would you like to add        anything else?    -   Ok I've added [QUANTITY] [SHORT TITLE] to your cart, would you        like to add anything else?    -   Ok, I added [MEDIUM TITLE]. Would you like to add anything else?        The bracketed text indicates versions of normalized entities        associated with the specific response templates. For example,        each of the response templates includes a [SHORT TITLE] or a        [MEDIUM TITLE] indicating a position in the template which a        product name must be inserted. The short title can be a short        form of the product name (e.g., “milk”), the medium title is a        medium form of the product name (e.g., “Great Value 2% milk”)        and a long title can be a long form of a product (e.g., “Great        Value 2% Milk, 1 Gallon”). As another example, a product        information response template may include, but is not limited        to:    -   Ok, I can get you a/an [QUANTITY] [UNIT] of [MEDIUM TITLE] for        [PRICE]. Would you like to add this to your cart?        The normalized entities of the product information template        include a quantity entity (identifying the amount of a product        in each unit), a unit entity (identifying the unit of sale for        the product), the medium title, and a price entity (identifying        the price of the product). Similarly, and as yet another        example, a set of templates for a product inquiry in a produce        domain (e.g., grocery domain) may include, but are not limited        to:    -   Ok, I can get you [MEDIUM TITLE] sold by [BY UNIT] for [PRICE].        Our average order size is [AVG NUM] [SHORT TITLE], would you        like to add this to your cart?    -   Ok, I can get you [MEDIUM TITLE] sold by [BY UNIT] for [PRICE].        Our average order size is [AVG NUM] [UNIT], would you like to        add this to your cart?    -   Ok, I can get you [MEDIUM TITLE] sold by [BY UNIT] for [PRICE].        Our average order size is [AVG NUM] [UNIT], which is about [AVG        COUNT] [SHORT TITLE]. Would you like to add this to your cart?        As can be seen in the produce domain example, the available        response templates may be selected based on available entities.        For example, the first response template may be used when a        produce product is sold individually, the second response        template may be used when there is no known average count [AVG        COUNT] but there is an average number [AVG NUM] added to a cart,        and the third response template may be used when an average        count is available. It will be appreciated that the proceeding        response templates are provided only as examples. The response        generation component 160 a can be configured with any response        template for any intent based on user experience considerations,        domain consideration, environmental considerations, etc.

In some embodiments, a voice store 722 is configured to maintain voicefiles and/or rules for converting the completed response template into asystem utterance for transmission to the user system 52 and conversionto audio. For example, in the illustrated embodiment, the voice store722 includes a plurality of voice templates and rules generated during avoice title generation process 724 by one or more designer systems 56and/or engineer systems 58. The voice title generation process 724generates dialogue responses from a catalog of available voice termsand./or dialogues stored in a voice database 726.

After selecting a response template, the response generation task 702instantiates the selected template and provides the requiredtemplate-filling entities. In some embodiments, the generated responseincludes a prompt or request for additional information and/orconfirmation. For example, FIG. 13 illustrates a branch visualization750 for response generation in response to execution of a search intent.At a first time, to, the natural language conversation architecture 102a receives a request 752 from a user to “Find Milk.” The NLU engine 130a identifies a search intent and a product entity (e.g., milk) and thetask stitcher 150 a implements an intent flow to search for the productentity. The search produces one or more results, including a productidentified as “Great Value Reduced Fat 2% Milk.” At time t₁, theresponse engine 160 a instantiates a search response template togenerate a search response 754 including a statement indicating asuccessful search and a prompt for a specific intent, e.g., “I foundGreat Value Reduced Fat 2% Milk. Would you like to add to cart?”

At time t₂, the natural language conversation architecture 102 areceives a user response. The response can include an affirmative (e.g.,yes) response 756 or a negative (e.g., no) response 758. Although notillustrated, the user response may also include an unrelated responseindicating a new intent. If the user response is affirmative, thenatural language conversation architecture 102 a identifies anadd-to-cart intent, which is processed by the NLU engine 130 a, thecontext mining component 140 a, and the task stitcher 150 a to add theselected product (i.e., “Great Value Reduced Fat 2% Milk”) to a cartassociated with the user. The response generator 160 a instantiates anadd-to-cart success response 760 indicating a successful add-to-cartintent at time t₃, e.g., “Ok, I have added Great Value Reduced Fat 2%Milk, 1 gallon to your cart.” The dialogue tree 750 for responding to a“Find Milk” request is completed when the product is added to the cart.

If the use response is negative, the natural language conversationarchitecture 102 a identifies a new product intent, which selects thenext product identified during the previously processed search intent,e.g., “Great Value Vitamin D Milk,” and instantiates a search responsetemplate to generate a second search response 762, e.g., “I have foundGreat Value Vitamin D Milk, 1 gallon for $3.04. Would you like to add tocart?” The response template randomly selected by the response generatorfor the second search response template is different than the firstsearch response template and includes the price of the identifiedproduct. It will be appreciated that the search response templateselected for the second identified product can be the same as and/ordifferent from the search response template selected for the firstidentified product.

At time t₄, the natural language conversation architecture 102 areceives a user response, e.g., an affirmative response 764, a negativeresponse 766, or an unrelated response (not shown). If an affirmativeresponse 764 is received, the the NLU engine 130 a and the contextmining component 140 a identify an add-to-cart intent and the taskstitcher 150 a executes an add-to-cart intent flow to add the selectedproduct (i.e., “Great Value Vitamin D Milk”) to a cart associated withthe user. The response generator 160 a instantiates an add-to-cartsuccess response template to generate a success response 768 at time t₅,e.g., “Ok, I have added Great Value Vitamin D Milk, 1 gallon to yourcart.” It will be appreciated that, although the successful add-to-cartresponse template selected for the second identified product is the sameas for the first product, the possible add-to-cart response templatesmay include one or more additional response templates that can beselected by the response generation component 160 a. The dialogue tree750 for responding to a “Find Milk” request is completed when theproduct is added to the cart.

If a negative response 766 is received, the natural languageconversation architecture 102 a may continue to select productsidentified by a prior search and present the products to a user forselection. After the user has rejected a predetermined number ofproducts, the natural language conversation architecture 102 aidentifies a terminal intent, indicating that the natural languageinteraction should be completed and the user directed to additionalresources. For example, in the illustrated embodiment, after presentingtwo rejected products, the natural language conversation architecture102 a identifies a terminal intent and the response generator 160 ainstantiates a terminal response template at time t₅ to generate aterminal response 770, e.g., “Sorry I couldn't find what you werelooking for.” Open the Grocery app for more results.” In variousembodiments, the terminal response template may be selected based on theadditional resource or resources available to the user.

The foregoing outlines features of several embodiments so that thoseskilled in the art may better understand the aspects of the presentdisclosure. Those skilled in the art should appreciate that they mayreadily use the present disclosure as a basis for designing or modifyingother processes and structures for carrying out the same purposes and/orachieving the same advantages of the embodiments introduced herein.Those skilled in the art should also realize that such equivalentconstructions do not depart from the spirit and scope of the presentdisclosure, and that they may make various changes, substitutions, andalterations herein without departing from the spirit and scope of thepresent disclosure.

What is claimed is:
 1. A system, comprising: a computing deviceconfigured to: identify an intent associated with at least one semanticelement of a user utterance; identify a missing semantic element in anintent flow associated with the identified intent by reviewing at leastone context entry in a set of context entries, the set of contextentries comprising a sequentially-stored structured representation ofeach utterance received from a user associated with the user utterance;generate the intent flow including the identified missing semanticelement; execute the intent flow associated with the identified intent;receive a request to execute the intent flow associated with theidentified intent; select a domain-specific intent flow from a set ofintent flows associated with the identified intent; execute a first taskof the domain-specific intent flow, wherein the first task generates afirst output; and execute a second task of the domain-specific intentflow, wherein the second task is selected based on the first output, andwherein the second task generates a second output.
 2. The system ofclaim 1, wherein a task stitcher is configured to receive one or moredomain-specific intent flows from a designer system associated with adomain.
 3. The system of claim 1, wherein the first task comprises anentity check configured to determine if a first required entity isidentified.
 4. The system of claim 1, wherein a system utterance isgenerated by a response generator component configured to implement aresponse generator task.
 5. The system of claim 4, wherein the responsegenerator task is configured to select a response template from aplurality of response templates associated with the identified intent.6. The system of claim 1, wherein each context entry in the set ofcontext entries comprises a root intent and an entity list, the rootintent comprising an intent specified in a user utterance and the entitylist identifies a product entity associated with the corresponding rootintent.
 7. A method comprising: identifying an intent associated with atleast one semantic element of a user utterance; identifying a missingsemantic element in an intent flow associated with the identified intentby reviewing at least one context entry in a set of context entries, theset of context entries comprising a sequentially-stored structuredrepresentation of each utterance received from a user associated withthe user utterance; generating the intent flow including the identifiedmissing semantic element; executing the intent flow associated with theidentified intent; receiving a request to execute the intent flowassociated with the identified intent; selecting a domain-specificintent flow from a set of intent flows associated with the identifiedintent; executing a first task of the domain-specific intent flow,wherein the first task generates a first output; and executing a secondtask of the domain-specific intent flow, wherein the second task isselected based on the first output, and wherein the second taskgenerates a second output.
 8. The method of claim 7, wherein a taskstitcher is configured to receive one or more domain-specific intentflows from a designer system associated with a domain.
 9. The method ofclaim 7, wherein the first task comprises an entity check configured todetermine if a first required entity is identified.
 10. The method ofclaim 7, wherein a system utterance is generated by a response generatorcomponent configured to implement a response generator task.
 11. Themethod of claim 10, wherein the response generator task is configured toselect a response template from a plurality of response templatesassociated with the identified intent.
 12. The method of claim 7,wherein each context entry in the set of context entries comprises aroot intent and an entity list, the root intent comprising an intentspecified in a user utterance and the entity list identifies a productentity associated with the corresponding root intent.
 13. Anon-transitory computer readable medium having instructions storedthereon, wherein the instructions, when executed by a processor cause atleast one device to perform operations comprising: identifying an intentassociated with at least one semantic element of a user utterance;identifying a missing semantic element in an intent flow associated withthe identified intent by reviewing at least one context entry in a setof context entries, the set of context entries comprising asequentially-stored structured representation of each utterance receivedfrom a user associated with the user utterance; generating the intentflow including the identified missing semantic element; executing theintent flow associated with the identified intent; receiving a requestto execute the intent flow associated with the identified intent;selecting a domain-specific intent flow from a set of intent flowsassociated with the identified intent; executing a first task of thedomain-specific intent flow, wherein the first task generates a firstoutput; and executing a second task of the domain-specific intent flow,wherein the second task is selected based on the first output, andwherein the second task generates a second output.
 14. Thenon-transitory computer readable medium of claim 13, wherein a taskstitcher is configured to receive one or more domain-specific intentflows from a designer system associated with a domain.
 15. Thenon-transitory computer readable medium of claim 13, wherein the firsttask comprises an entity check configured to determine if a firstrequired entity is identified.
 16. The non-transitory computer readablemedium of claim 13, wherein a system utterance is generated by aresponse generator component configured to implement a responsegenerator task, the response generator task configured to select aresponse template from a plurality of response templates associated withthe identified intent.
 17. The non-transitory computer readable mediumof claim 13, wherein each context entry in the set of context entriescomprises a root intent and an entity list, the root intent comprisingan intent specified in a user utterance and the entity list identifies aproduct entity associated with the corresponding root intent.